Recommender System for Songs


In [1]:
import graphlab
%matplotlib inline


This non-commercial license of GraphLab Create for academic use is assigned to william_gray@alumni.brown.edu and will expire on March 20, 2018.
[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: /tmp/graphlab_server_1491953199.log

Download data, which contains data on songs, users and listen counts.


In [2]:
URL = 'https://d396qusza40orc.cloudfront.net/phoenixassets/song_data.csv'
#use SFrame for larger dataset
song_data = graphlab.SFrame(URL)
song_data.head()


Downloading https://d396qusza40orc.cloudfront.net/phoenixassets/song_data.csv to /var/tmp/graphlab-williamgray1/17010/5329edc6-3dff-4624-ab72-66ed498ac085.csv
Finished parsing file https://d396qusza40orc.cloudfront.net/phoenixassets/song_data.csv
Parsing completed. Parsed 100 lines in 1.45803 secs.
------------------------------------------------------
Inferred types from first 100 line(s) of file as 
column_type_hints=[str,str,int,str,str,str]
If parsing fails due to incorrect types, you can correct
the inferred type list above and pass it to read_csv in
the column_type_hints argument
------------------------------------------------------
Read 374317 lines. Lines per second: 149913
Finished parsing file https://d396qusza40orc.cloudfront.net/phoenixassets/song_data.csv
Parsing completed. Parsed 1116609 lines in 4.35139 secs.
Out[2]:
user_id song_id listen_count title artist
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOAKIMP12A8C130995 1 The Cove Jack Johnson
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOBBMDR12A8C13253B 2 Entre Dos Aguas Paco De Lucia
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOBXHDL12A81C204C0 1 Stronger Kanye West
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOBYHAJ12A6701BF1D 1 Constellations Jack Johnson
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SODACBL12A8C13C273 1 Learn To Fly Foo Fighters
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SODDNQT12A6D4F5F7E 5 Apuesta Por El Rock 'N'
Roll ...
Héroes del Silencio
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SODXRTY12AB0180F3B 1 Paper Gangsta Lady GaGa
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOFGUAY12AB017B0A8 1 Stacked Actors Foo Fighters
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOFRQTD12A81C233C0 1 Sehr kosmisch Harmonia
b80344d063b5ccb3212f76538
f3d9e43d87dca9e ...
SOHQWYZ12A6D4FA701 1 Heaven's gonna burn your
eyes ...
Thievery Corporation
feat. Emiliana Torrini ...
song
The Cove - Jack Johnson
Entre Dos Aguas - Paco De
Lucia ...
Stronger - Kanye West
Constellations - Jack
Johnson ...
Learn To Fly - Foo
Fighters ...
Apuesta Por El Rock 'N'
Roll - Héroes del ...
Paper Gangsta - Lady GaGa
Stacked Actors - Foo
Fighters ...
Sehr kosmisch - Harmonia
Heaven's gonna burn your
eyes - Thievery ...
[10 rows x 6 columns]

Data exploration


In [3]:
len(song_data)


Out[3]:
1116609

In [4]:
# make graphs inline
graphlab.canvas.set_target('ipynb')
# histogram of top songs
song_data['song'].show()



In [5]:
# unique users
users = song_data['user_id'].unique()
len(users)


Out[5]:
66346

Create a simple recommender based on song popularity


In [6]:
# create training and test data with an 80/20 random split
train_data, test_data = song_data.random_split(.8, seed=0)

In [7]:
popularity_model = graphlab.popularity_recommender.create(train_data,
                                                          user_id='user_id',
                                                          item_id='song')


Recsys training: model = popularity
Warning: Ignoring columns song_id, listen_count, title, artist;
    To use one of these as a target column, set target = 
    and use a method that allows the use of a target.
Preparing data set.
    Data has 893580 observations with 66085 users and 9952 items.
    Data prepared in: 1.65671s
893580 observations to process; with 9952 unique items.

In [8]:
# model only uses song popularity to make recommendation
popularity_model.recommend(users=[users[0]])


Out[8]:
user_id song score rank
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Sehr kosmisch - Harmonia 4754.0 1
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Undo - Björk 4227.0 2
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
You're The One - Dwight
Yoakam ...
3781.0 3
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Dog Days Are Over (Radio
Edit) - Florence + The ...
3633.0 4
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Revelry - Kings Of Leon 3527.0 5
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Horn Concerto No. 4 in E
flat K495: II. Romance ...
3161.0 6
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Secrets - OneRepublic 3148.0 7
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Fireflies - Charttraxx
Karaoke ...
2532.0 8
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Tive Sim - Cartola 2521.0 9
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Drop The World - Lil
Wayne / Eminem ...
2053.0 10
[10 rows x 4 columns]


In [9]:
# recommendations will be the same for every user. This is similar to 'Most Emailed' section of New York Times
popularity_model.recommend(users=[users[1]])


Out[9]:
user_id song score rank
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Sehr kosmisch - Harmonia 4754.0 1
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Undo - Björk 4227.0 2
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
You're The One - Dwight
Yoakam ...
3781.0 3
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Dog Days Are Over (Radio
Edit) - Florence + The ...
3633.0 4
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Revelry - Kings Of Leon 3527.0 5
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Horn Concerto No. 4 in E
flat K495: II. Romance ...
3161.0 6
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Secrets - OneRepublic 3148.0 7
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Hey_ Soul Sister - Train 2538.0 8
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Fireflies - Charttraxx
Karaoke ...
2532.0 9
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Tive Sim - Cartola 2521.0 10
[10 rows x 4 columns]

More personalized song recommendation model


In [10]:
personalized_model = graphlab.item_similarity_recommender.create(train_data,
                                                        user_id='user_id',
                                                        item_id='song')


Recsys training: model = item_similarity
Warning: Ignoring columns song_id, listen_count, title, artist;
    To use one of these as a target column, set target = 
    and use a method that allows the use of a target.
Preparing data set.
    Data has 893580 observations with 66085 users and 9952 items.
    Data prepared in: 1.63973s
Training model from provided data.
Gathering per-item and per-user statistics.
+--------------------------------+------------+
| Elapsed Time (Item Statistics) | % Complete |
+--------------------------------+------------+
| 2.716ms                        | 1.5        |
| 66.332ms                       | 100        |
+--------------------------------+------------+
Setting up lookup tables.
Processing data in one pass using dense lookup tables.
+-------------------------------------+------------------+-----------------+
| Elapsed Time (Constructing Lookups) | Total % Complete | Items Processed |
+-------------------------------------+------------------+-----------------+
| 383.08ms                            | 0                | 0               |
| 1.77s                               | 100              | 9952            |
+-------------------------------------+------------------+-----------------+
Finalizing lookup tables.
Generating candidate set for working with new users.
Finished training in 2.87088s

In [11]:
personalized_model.recommend(users=[users[0]])


Out[11]:
user_id song score rank
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Cuando Pase El Temblor -
Soda Stereo ...
0.0194504536115 1
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Fireflies - Charttraxx
Karaoke ...
0.0144737317012 2
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Love Is A Losing Game -
Amy Winehouse ...
0.0142865960415 3
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Marry Me - Train 0.014133471709 4
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Secrets - OneRepublic 0.013591665488 5
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Sehr kosmisch - Harmonia 0.0133987894425 6
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Te Hacen Falta Vitaminas
- Soda Stereo ...
0.0129302831796 7
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
OMG - Usher featuring
will.i.am ...
0.0127778282532 8
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
Y solo se me ocurre
amarte (Unplugged) - ...
0.0123411279458 9
c66c10a9567f0d82ff31441a9
fd5063e5cd9dfe8 ...
No Dejes Que... -
Caifanes ...
0.0121042499175 10
[10 rows x 4 columns]


In [12]:
personalized_model.recommend(users=[users[1]])


Out[12]:
user_id song score rank
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Riot In Cell Block Number
Nine - Dr Feelgood ...
0.0374999940395 1
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Sei Lá Mangueira -
Elizeth Cardoso ...
0.0331632643938 2
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
The Stallion - Ween 0.0322580635548 3
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Rain - Subhumans 0.0314159244299 4
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
West One (Shine On Me) -
The Ruts ...
0.0306771993637 5
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Back Against The Wall -
Cage The Elephant ...
0.0301204770803 6
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Life Less Frightening -
Rise Against ...
0.0284431129694 7
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
A Beggar On A Beach Of
Gold - Mike And The ...
0.0230024904013 8
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Audience Of One - Rise
Against ...
0.0193938463926 9
279292bb36dbfc7f505e36ebf
038c81eb1d1d63e ...
Blame It On The Boogie -
The Jacksons ...
0.0189873427153 10
[10 rows x 4 columns]


In [13]:
personalized_model.get_similar_items(['With Or Without You - U2'])


Out[13]:
song similar score rank
With Or Without You - U2 I Still Haven't Found
What I'm Looking For ...
0.042857170105 1
With Or Without You - U2 Hold Me_ Thrill Me_ Kiss
Me_ Kill Me - U2 ...
0.0337349176407 2
With Or Without You - U2 Window In The Skies - U2 0.0328358411789 3
With Or Without You - U2 Vertigo - U2 0.0300751924515 4
With Or Without You - U2 Sunday Bloody Sunday - U2 0.0271317958832 5
With Or Without You - U2 Bad - U2 0.0251798629761 6
With Or Without You - U2 A Day Without Me - U2 0.0237154364586 7
With Or Without You - U2 Another Time Another
Place - U2 ...
0.0203251838684 8
With Or Without You - U2 Walk On - U2 0.0202020406723 9
With Or Without You - U2 Get On Your Boots - U2 0.0196850299835 10
[10 rows x 4 columns]

Quantitative model performace comparison


In [14]:
model_performace = graphlab.recommender.util.compare_models(test_data,
                                                           [popularity_model, personalized_model],
                                                           user_sample=0.05)


compare_models: using 2931 users to estimate model performance
PROGRESS: Evaluate model M0
recommendations finished on 1000/2931 queries. users per second: 9026.82
recommendations finished on 2000/2931 queries. users per second: 10354.9
WARNING:root:Model trained without a target. Skipping RMSE computation.
Precision and recall summary statistics by cutoff
+--------+-----------------+------------------+
| cutoff |  mean_precision |   mean_recall    |
+--------+-----------------+------------------+
|   1    | 0.0255885363357 | 0.00729273664279 |
|   2    | 0.0266120777892 |  0.016144499384  |
|   3    | 0.0234277266007 | 0.0201149318857  |
|   4    | 0.0232855680655 | 0.0262210163694  |
|   5    | 0.0215626066189 | 0.0300954703206  |
|   6    | 0.0206982827249 | 0.0344427224914  |
|   7    | 0.0194960276844 | 0.0371904757494  |
|   8    | 0.0184663937223 | 0.0401322134915  |
|   9    | 0.0174760225937 |  0.042171718397  |
|   10   | 0.0169907881269 | 0.0460115004353  |
+--------+-----------------+------------------+
[10 rows x 3 columns]

PROGRESS: Evaluate model M1
recommendations finished on 1000/2931 queries. users per second: 7077.69
recommendations finished on 2000/2931 queries. users per second: 8891.69
WARNING:root:Model trained without a target. Skipping RMSE computation.
Precision and recall summary statistics by cutoff
+--------+-----------------+----------------+
| cutoff |  mean_precision |  mean_recall   |
+--------+-----------------+----------------+
|   1    |  0.202320027294 | 0.066105288289 |
|   2    |  0.163766632549 | 0.101820840791 |
|   3    |  0.141817354714 | 0.128004752031 |
|   4    |  0.126066189014 | 0.148255462123 |
|   5    |  0.114568406687 | 0.16514124513  |
|   6    |  0.104003184351 | 0.178890982446 |
|   7    | 0.0969927377297 |  0.192544745   |
|   8    | 0.0906687137496 | 0.204304139743 |
|   9    | 0.0849918495773 | 0.214947362285 |
|   10   | 0.0802115319004 | 0.224407998127 |
+--------+-----------------+----------------+
[10 rows x 3 columns]

The average precision and recall of model M1 (personalized model) are much higher than model M0 (popularity model)

Deeper look at the recommendations

First, look at top artists by listen count


In [16]:
song_data.groupby(key_columns='artist', operations={'total_count': graphlab.aggregate.SUM('listen_count')}).sort('total_count', ascending=False)


Out[16]:
artist total_count
Kings Of Leon 43218
Dwight Yoakam 40619
Björk 38889
Coldplay 35362
Florence + The Machine 33387
Justin Bieber 29715
Alliance Ethnik 26689
OneRepublic 25754
Train 25402
The Black Keys 22184
[3375 rows x 2 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.


In [17]:
# for quicker analysis, look at first 10000 unique users
subset_test_users = test_data['user_id'].unique()[0:10000]

In [22]:
# Top recommendation for each user in the subset
top_rec = personalized_model.recommend(subset_test_users,k=1)


recommendations finished on 1000/10000 queries. users per second: 6788.68
recommendations finished on 2000/10000 queries. users per second: 8735.15
recommendations finished on 3000/10000 queries. users per second: 10057.3
recommendations finished on 4000/10000 queries. users per second: 10791.4
recommendations finished on 5000/10000 queries. users per second: 11211.2
recommendations finished on 6000/10000 queries. users per second: 11503.9
recommendations finished on 7000/10000 queries. users per second: 11838.5
recommendations finished on 8000/10000 queries. users per second: 12071.7
recommendations finished on 9000/10000 queries. users per second: 12337.5
recommendations finished on 10000/10000 queries. users per second: 12236.8

In [26]:
# most recommended song in subset
recommendation_count = top_rec.groupby(key_columns='song', operations={'total count': graphlab.aggregate.COUNT('song')})
recommendation_count.sort('total count', ascending=False)


Out[26]:
song total count
Undo - Björk 433
Secrets - OneRepublic 352
Revelry - Kings Of Leon 229
You're The One - Dwight
Yoakam ...
166
Fireflies - Charttraxx
Karaoke ...
111
Hey_ Soul Sister - Train 110
Sehr kosmisch - Harmonia 105
Horn Concerto No. 4 in E
flat K495: II. Romance ...
96
OMG - Usher featuring
will.i.am ...
64
Dog Days Are Over (Radio
Edit) - Florence + The ...
46
[3145 rows x 2 columns]
Note: Only the head of the SFrame is printed.
You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.

As expected, the top recommeded songs come from the most popular artists by listen count.


In [ ]: